JBoss Community Archive (Read Only)

RHQ 4.5

Code coverage tools

Overview

Code coverage describes the degree to which the source code of a program has been tested.
The basic information about code coverage can be found at http://www.bullseye.com/coverage.html .
Code coverage runs in most frameworks in these phases :

The main java projects which are working or worked on code coverage are :

Selection

The criteria for selection of right tool for our projects are:

  • Maven integration

  • Hudson integration

  • Reporting in HTML

  • Fast processing

  • Good documentation

  • Progress on project support and development

The mentioned tools will be divided into there groups : 
a) Tools that don't fullfill criteria

  • dead tools
    -JVMDI, JBLANKET, InsectJ, noUnit, Hansel, gretel, Groboutils, quilt

  • different testing method
    -jester

  • other
    -codecover, coverclipse

  • licensed
    Jcoverage
    Clover

b) Tools that fullfils criteria

  • Clover

  • Cobertura

  • Emma

a) Tools that don't fullfill criteria

  • Dead tools

QUILT

Source

http://quilt.sourceforge.net/

Description

Quilt is a Java software development tool that measures coverage , the extent to which unit testing exercises the software under test.
Optimized for use with the JUnit unit test package

Start

Ant, Maven

Technology

statement coverage

Frameworks

BCEL

Reports

 

Report example

 

Last release

20 October 2003

License

Apache License, Artistic license

IDE support

No

GROBOUTILS

Source

http://groboutils.sourceforge.net/

Description

The popular tools included with GroboUtils include multi-threaded tests, hierarchial unit tests, and a code coverage tool.

Start

Ant

Technology

logging statements for tracking coverage

Frameworks

BCEL

Reports

XML, xsl-t transformations, HTML

Reports example

http://groboutils.sourceforge.net/codecoverage/CoverageReport-LineCount.html

Last release

2004-11-05

Licence

MIT licence

Additional information

  • well documented

  • instruments ear files, war files, zip files within zip file

  • slower performance

GRETEL

Source

http://www.cs.uoregon.edu/research/perpetual/dasada/Software/Gretel/

Description

Gretel is a test coverage monitoring tool for Java programs. The current version provides statement
coverage monitoring (identifying which lines of Java have been executed, and which have not been touched by testing).

Start

GUI, command line

Technology

 

Frameworks

BCEL

Reports

GUI

Last realease

June 2002

License

BSD

HANSEL

Source

http://hansel.sourceforge.net/

Description

Hansel is an extension to JUnit that adds code coverage testing to the testing framework.
Much of the source code is taken from Gretel a "Residual Test Coverage Monitoring"-Tool developed at the University of Oregon.

Start

ANT

Technology

 

Frameworks

ASM

Reports

 

Last release

2006-10-04

License

BSD

Aditional Information

NOUNIT

Source

http://nounit.sourceforge.net/

Description

NoUnit allows you to see how good your JUnit tests are. It generates a report from your code to graphically show you how many of your project's methods are being tested , and how well

Start

command line

Reports

XML, xsl, HTML

Report example

http://nounit.sourceforge.net/sample.html

Last release

2003-12-15

License

GNU GPL

IDE support

 

Addition information

dont need source codes

  • Different technology

JESTER

Source

http://jester.sourceforge.net/

Description

Jester doesn't watch which lines of code have been executed.
Instead Jester changes the source code, recompiles it, and runs the test suite to see if anything breaks.

Start

 

Technology

mutation testing

Frameworks

 

Reports

easy HTML

Report example

http://jester.sourceforge.net/jester.html

Last release

jester 2005-04-02, simple-jester 2009-11-24

License

open license (http://jester.sourceforge.net/license.html)

IDE support

http://www.dcs.shef.ac.uk/intranet/teaching/projects/archive/ug2005/abs/u2sl.htm
Undergraduate project dissertation
there is another eclipse plugin ? http://jester.cvs.sourceforge.net/viewvc/jester/jester-eclipse-plugin/

Additional information

  • very poor documentation, no comments in code

  • there is also Pester for Python PyUnit tests

  • Other

COVERLIPSE

Source

http://coverlipse.sourceforge.net/

Description

Coverlipse is an Eclipse plugin that visualizes the code coverage of JUnit Tests.

Start

Eclipse

Technology

Statement Coverage

Frameworks

 

Reports

Eclipse, java editor

Reports example

http://coverlipse.sourceforge.net/screenshots.php

License

CPL

Last release

02-14-2009

Comments

only eclipse support

CODECOVER

Source

http://codecover.org/

Description

 

Start

Eclipse, Command line, ant

Technology

CodeCover measures statement, branch, loop, term coverage (subsumes MC/DC), question mark operator coverage, and synchronized coverage

Frameworks

 

Reports

template engine Velocity, html, csv

Report example

http://codecover.org/documentation/tutorials/SimpleJavaAppReport.html

Last release

19.04.2010

Licence

Eclipse Public Licence (EPL)

Comments

- there is no maven support
- nice reports
- good documentation

  • Licensed
    JCoverage
    Clover

b) Tools that fullfills criteria

CLOVER

Source

http://www.atlassian.com/software/clover/

Description

Clover delivers actionable Java code coverage metrics to assess the impact of your tests. It goes beyond raw test coverage to help you focus your limited testing resources where they will be most effective.

Start

Eclipse, Netbeans, IntelliJ Command line, ant, maven

Technology

Instrumentation of source codes

Frameworks

 

Reports

html, xml, pdf, json

Report example

http://www.atlassian.com/software/clover/sample-reports.jsp

Last release

9-September-2009

Licence

Commercial, sale for acamedics, free for non-profit and open source projects

Comments

- Most complete reports and statistics
- integration with other atlassian products
- good documentation
- Distributed Test Coverage Aggregation
- Extensibility via java api
- Hudson plugin

COBERTURA

Source

http://cobertura.sourceforge.net/

Description

Cobertura is a free Java tool that calculates the percentage of code accessed by tests.
It can be used to identify which parts of your Java program are lacking test coverage.
It is based on jcoverage.

Start

Ant,Command line, Maven

Technology

McCabe cyclomatic code complexity of each class (http://en.wikipedia.org/wiki/Cyclomatic_complexity)

Frameworks

ASM

Reports

Can generate reports in HTML or XML.

Report example

http://cobertura.sourceforge.net/sample/

Last release

last version 1.9.4.1 released 2010-03-03

License

GNU GPL 2

IDE support

http://ecobertura.johoop.de/
written in scala,
supported by one man
http://www.eclipse-plugins.info/eclipse/plugin_details.jsp?id=1151
added 3,2005 modified sep 13 2006

Additional Information

  • cobertura plugin for hudson http://wiki.hudson-ci.org/display/HUDSON/Cobertura+Plugin

  • is a fork of jcoverage started at 2005

  • When instrumenting, you can now specify a zip, jar, war,ear or sar file and Cobertura will instrument any classes inside of the archive and archives within archives

  • Ability to instrument classes on a given classpath instead of specifying filesets

  • Can now use multiple filesets in the cobertura-instrument task

  • support for classes compiled with AspectJ

  • Support for instrumenting classes written in Groovy

  • thread safe

  • data obtained in different instrumentation or test runs can be merged together

Comment

 

EMMA

Source

http://emma.sourceforge.net/ http://emma.sourceforge.net/

Description

EMMA is an open-source toolkit for measuring and reporting Java code coverage. EMMA distinguishes itself from other
tools by going after a unique feature combination: support for large-scale enterprise software development while keeping individual
developer's work fast and iterative.

Start

ANT , EMMA maven plugin ( The maven EMMA plugin simply wraps the EMMA Ant tasks, so it behaves in a very similar way),

Technology

 

Reports

plain text, HTML (source code linking), XML

Report example

http://emma.sourceforge.net/coverage_sample_a/index.html http://emma.sourceforge.net/coverage_sample_a/index.html

Support

No write transaction on CVS during last year

License

CPL

IDE support

support for eclipse EclEmma (http://sourceforge.net/projects/eclemma/ http://sourceforge.net/projects/eclemma/ )

Comments

  • should be fast

  • there is a hudson plugin

  • eclipse support

  • Coverage data obtained in different instrumentation or test runs can be merged together.

Summary

From mentioned frameworks Clover provides the best services. Clover has the best reports, good documentation and easy to use configuration.
As a open source project we can use the product for free. To get the free license the project must be registred and approved by atlassian.
There is also term in license that code coverage reports must be publicly accessible.

Emma provides good enough functionality but the last release was at 2005-07-27 and there is no progress on the project anymore.

Cobertura provides all functions which are needed for code coverage. The main difference between Clover and Cobertura is in reports.
Cobertura report task provides the same basic data but clover offers much more statistics about projects and coverage.
Cobertura maven plugin has also problems with coverage of integration tests, but there is a workaround for this problem.

I vote for Cobertura because it is open source project and provides sufficient data about code coverage of our project.

(Interesting tool is also Jester which uses mutation testing, but it is not what we are looking for right now.)

JBoss.org Content Archive (Read Only), exported from JBoss Community Documentation Editor at 2020-03-12 12:55:38 UTC, last content change 2010-10-07 06:01:13 UTC.